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ABSTRACT 

NLS-spHOLAR is a prototype system that uses 
artificial intelligence techniques to teach computer -naive people how 
to use a powerful and complex editor^ This new kind of computer " 
assisted instruction system integrates systematic teaching with 
actual practice, keeping the user under tutorial supervision while 
allowing him to try out what he learns on the editor. NLS-SCHOLAR can 
be used as an on-line help system, outside the tutorial environment, 
in the course ^f a user*s actual work. Testing on this system 
revealed that the teaching component of NIS-SCHOLAR is very 
effective, but the system*s performance as an on-line help fap.lity 
needs improvement* Techniques used in NLS-SCHOLAR are general and can 
be applied to the teaching of a wide variety of computer related 
activities. (Author/CH) 



♦ Documents acquired by ERIC include many informal unpublished ♦ 
♦"Materials not available from other sources. ERIC makes every effort ♦ 

♦ to obtain the best copy available. Nevertheless, items of marginal * 

♦ reproducibility are often encountered and this affects the quality ♦ 

♦ of the microfiche and hardcopy reproductions ERIC makes available ♦ 

♦ vicb the ERIC, Document Reproduction , Service (EDRS)#. EDRS is not ♦ 

♦ responsible for the quality of the original document. Reproductions ♦ 

♦ supplied by EDRS are the best that can 'be made from the original. ♦ 
♦♦*♦♦♦♦♦♦♦*♦♦♦♦*♦♦♦♦♦♦**♦**********♦*♦♦♦♦ ♦♦^ 



ERIC 



1 1 



/ 



oo 

<^ 

OO 

4 



BSD-TR-75-358 



NLS-SCHOLAR: MODIFICATIONS AND FIELD TESTING 



1 1 



Boft, Beranek and Newman, Inc. 
50 MouFton Street 
Cambridge, MA 02138 



November 1975 



U.S. DEPARTMENTOF HEALTH, > 
EDUCATIQaAHWElH«AHE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS REJirVED FROM 
Ir.l^^.5^°'^ ^'^ ORGANIZATION ORIGIN* 
ATINGIT ROINTS OF VIEW OR OPINIONS 
lltl^^ ^ NECESSARILY REPRE- 

SENT OFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLICY 



Approved for Public Refeose; 
Distribution Unlimited. 



Prepared for and Sponsored by 

DEPUTY FOR COMAAAND AND MANAGEMENT SYSTEMS 
ELECTRONIC SYSTEMS DIVISION 
HANSCOM AIR FORCE BASE, MA 01731 

DEFENSE APVANCED RESEARCH PROJECTS AGENCY 
1400 WILSON BOULEVARD 
ARLINGTON, VA ^209 
ARPA Order No. 2984 



The views and concfusions contained in this document are those of the authors 
and shoufd not be interpreted as necessariFy representing the officiaf policies, 
either expressed or impFied, of the Defense Advanced Research Projects Agency 
or the U. S. Government, 



LEGAL NOTICE 

When U.S. Government drawings, specifications or other data are used for any 
purpose other than a definitely related government procurement operation, the 
government thereby incurs no responsibility nor any obligation whatsoever; and 
the fact that the government may hove formulated, furnished, or in any way sup* 
plied the soid drawings, specifications, or other data is not to be regarded by 
implication or otherwise as in any manner licensing the holder or any other person 
or conveying any rights or permission to manufacture, use, or sail any patented 
invention that may in any way be related thereto. 



OTHER NOTICES 

n 

Do not return this copy. Retain or destroy, 

< 

"This technical report has been reviewed and is approved for publication 



SYLVIA R. MAYER, GS-U 
Project Scientist 

FOR THE COMMANDER 




FRANK J. EMM/^ Colonel, USAF 
Director, Information Systems 
•Technology Applications Office 
Dept^y for Command & Management Systems 



Unclassified 



SECURITY CLASSIFICATION OF THIS PAGE (Wh^n D«)« Bntermd) 



REPORT DOCUMENTATION PAGE 



READ INSTRUCTIONS 
BEFORE COMPLETING FORM 



1. REPORT NUMBER 

ESD-TR-75-358 



2. govt'accession no 



3. RECIPIENT'S CATALOG NUMBER 



TITLE (md Subtlit^) 

NIS-SCHOLAR: MODIFICATIONS 
,ANbT4ELD TESTING 



5. TYPE OF REPORT 4 PERIOD COVERED 



«. PERFORMING ORG. REPORT NUMBER 



7. 



|THORf«; 

\ario C. Grignetti 
iaura Gould 

iatherine Hausmann, et aF 



8. CONTRACT OR GRANT NUMBERf*; 

Fr9628-75-C-0r59 
ARPA Order 2984 



PERF0RMIN<5 ORGANIZATION NAME AND ADDRESS 

Baft, Beranek and Newman, |nc. 
50\Moufton Street 
Cambridge, MA 02138 



10. PROGRAM ELEMENT. PROJECT. TASK 
AREA a WORK UNIT NUMBERS 

62706E Progj^am Element 



1 1. COI^TROLLING OFFICE NAME AND AboRESS 

Deputy for Commancj and Management Systems 
Electronic Systems Division 
Hanshom Air Force Base. AAA 0f73f 



12. REPORT DaVE 

November F975 



13. NUMBER OF PAGES 
77 



14 MONITORING AGENCY NAME ft AOO^ESS(tt dlttermt from Conttoitini Otfiee) 

Defense Advanced Research) Projects Agency 
1400 Wilson Boulevard 
Arlington, VA 22209 , 



IS. SECURITY CLASS. ^0/ thiB reporij 

UNCLASSIFIED 



1S«. DECLASSIFICATION/ DOWNGRADING 
SCHEDULE 



16. DISTRIBUTION STATEMENT (ot thiu R^potl) 

ApprovefJ for pubfic release; distribution unlimited. 



17. DtSTRlBUTld^N STATEMENT (ot th9 mb»ttmet Tiffd tn Block 20, It dltUront from R9pori) 



16. SUPPLEMENTAlRY NOTES 



19* KEY WORDS (Contliw* on rwymfa 9ld% ii rfc*B%%ry and Idantity by block number) * 

Artificial Intelligence, Computer Assisted Instruction, Natural Language 
Processing, Semantic Grammar, Semantic Network, Tutorial Supervision, 
On-Line Assistai\ce, Question Answering 

\ 



20. ABSTRACT (Contiw ori, r«v«rt« titf* It n«c««fl«fy mnd IdmntHy by block number) 



NLS-SCHOLAR is a t>rototyf)e system that ^uses Artificial Intelligence techniques 
to teach computer -^^naive people how to u^e a powerful and complex editor. It 
represents a new kind of Computer Assisted Instruction (CAI) system that 
integrates systematic t,eaci;iing with actual practice, i.e., one which ckn keep 
the user under tutorial su^iervision while allowing him to try out what he 
learns on the systeik he is learning about. 



\ 




(over) 



O DD,!;r73l473 



EDITION OF 1 NQVeS^SOpSOL^TE . 



ERIC 



Unclas%rfied 



SECURITY CLAi^FiCAtlON OF THIS PAGE (Whan DaU Bnfrad) 



Unclassified 

SECURITY CLASSIFICATION 'OF THIS PAGE(W7i»n Dmta Enffd) 



20. (cont) 

NLS-SCHOLAR can also be used as an on-line help system outside the tutorial 
environment, in the tqurse of a user's actual work. This "capability of . 
combining on-line assistance with training is an extension of the traditional 
notion of CAI 

The system is now operational. Limited but realistic testing revealed that 
the teachings of NLS-SCHOLAR are very effective, and that the system's 
performance as an on-line help facility needs improvement. Most of the 
problems encountered are very easy to fix. 

The techniques used in NLS-SCHOLAR are general and can be applied to the 
teaching of a wide variety of computer related aSctivities. 



er|c 



SECURITY CLASSIFICATION OF THIS PAGEr»Wi«n D»tm Bnt%fd) 



^ TABLE OF CONTENTS t * 

Page 

SECTION I - INTRODUCTION 3 

Overall Approach 3 

Objectives ] 5 

Outline . . • 5 

SECTION II - DEVELOPMENTAL WORK 7 

Overview ^ 

The Control Structur-e 12 

Tutorial Material. . . 16 

New Text 16 

Branching ^ 17 

Tasks \ ^ ^ 2.8 

Questions. . . . 18 

Answers ^ 19 

English Front End 19 

The Parsing Process 

'^Fuzziness 26 

Instantiation of Varia-)les 27 

Further usefe^ of LISP-NIS to. answer questions . . 29 

Human Engineering Features 30 

Stop and Resume • 30 

Getting help from an expert 31 

Question mark - f , 32 

Efficiency > 33 

SECTION III - OPERATIONAL TESTING AND RESULTS . . . . ' 35 

f 

General Results ,^ , 35 

Overview - ' 38 

The "easy problems" 39 

Spelling errors. . : . / 39 

Unanticipated synonyms 40 

Common but unanticipated syntax. . . . ^ . . 41 

^Lack of knowledge / . . ' ' 41 

Poor answers !• ♦ • 42 

Unanticipated environments . . . .' - 43 

The Harder Problems 44 

First scenario 45 

Second scenario 48 

Third scenario 50 

Fourth scenario 54 

Fifth scenario 57 

SECTION IV - RECOMMENDATIONS AND CONCLUSIONS 62 

Epilogue ^ , 55 

APPENDIX 57 

Review of NLS-SCHOLAR by ISI . . 67 

Comments on the review 74 

REFERENCES 77 

/ 6 



SECTION I - INTRODUCTldN 

This is the Final Report on a six-month effort to 

improve and fierd test NLS-SCHOLAR> [Grignetti 1975], a CAI 

system that employs Artificial Intelligence techniques to 
teach people how to use the BASE subsystem of NLS.* 

This Report documents the ^Ita^es made to the August 
1974 versiQn.of NLS-SCHOLAR to prepare it for toting in the 
field, and docunents the results and conclusions obtained 
from this testing. Since NLS-SCHOLAR was developed under a 
previous contract, this rei^ort is conceived as an 
"incremental" one that should be read in conjunction with 
the Final Report [Grignetti 1974] on our previous effort. 

Overall Approach 

NLS-SCHOLAR is oriented towards teaching NLS to naive 
users, such as secretaries, v/ho have very limited experience 
with computer-based text editing systems. Therefore, its 
tutorial material is written assuming practically no 
knowledge of computer usage on the student's part; the 
necessary conceptual framework is built up from the most 

*BASE is the powerful editor of the oN«Line System (NLS), an 
in6reasingly used text manipulation system developed by 
Douglas Engelbart and his co-workers at the Augmentation 
Research Center of Stanford Research Institute. 



primitive notions, such as striking a key on a termina,! 
keyboard . 

The two basic pillars 6n which the system's approaeh^^ 
founded are: a) interactiveness and mixed-initiative, and b) 
supervised practice of the procedural knowledge being 



taught . \ ^ 

Interactiveness and mixed-initiative are necessary so 
that the student doesn't feel "caught" in a situation over 
which he has no control. The system is designed so that any 
time it is the student's turn to type, he can ask questions 
himself (instead of just answering the questions posed by 
the system) , or direct the system to perform certain actions 
for him ( like^ executing NLS commands expressed in English). 

Supervised practice is absolutely fundamental. Little 
knowledge about "how to ■ do" things can be taught by mere 
descr^iptions ; many procedures can only be taught by 
demonstration, and practice is essential. A supervised 
"hands on" environment is crucial to impressing newly 
acquired procedural knowledge in the student's mind. 
NLS-SCHOLAR provides such an environment by requesting 
students to perform NLS editing tasks using (what appears to 
them to be) the very system they are being taught about, by 
remaining "aware" of what they are doing, and by commenting 
* on their performance. 
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Objectives 

Our ultimate goal is to develop NLS-SCHOLAfi so that it 
can be used as an operational tool over the ARPA network, in 
support of the National Software Works (NSW) user's. The 
specific objectives of the work described in this document 
were: 

a) Expand and modify the NLS-SCHOLAR system as it existed at 
the end of its first year of development, incorporate 
features we perceived as needed, and correct known 
limitations 

h) Test the newly developed system in a limited but 

realistic operational environnent ^ 

« 

c) Use the feedback and experience obtained in the field to 
evaluate the system and to ^formulate plans for the next 
stage of modification and expansion 

These objectives have been achieved. 

Outline 

In Section II of the body of this report, we describe 
in detail the developmental work performed to achieve our 
objectives; in Section III we de^crJJ^e^Ui^^ obtained 
during field testing of the present version ofTjtS^^CHOLAR . 

9 



SECTION II - 




ELOPMENTAL WORK 



Ih this'-SeGtlon we-^describe thei Vork accomplished < to 
bring ^ NLS^^S^OLAR a state sufficiently stable and ro|)ust 
SQ^ that testing it^'ope rationally would Vield meaningful 
results. 




1) In order to provide the flexibility and modularity 
required to effect changes easily, we designed and 
implemented a new control structure that uses an 
implementation of the Bobrow/Wegbreit stack scheme for 
multiple environments ("spaghetti stacks") that is 

p^^ovided in the recently releaaed^LXS^-, [3obrow 1973, 



INTERLISP 1975]. 

« 

2) To increase the effectiveness of our tutorial material, 
we developed a prototype Agenda Language that allows us 
Jbo write English-like lessons incorporating branching, 
remedial loops, quizzes, etc. 

3) In order to provide a useful tutori^ng environment in 
spite of expected system limitations, we incorporated a 
fall-back mode wherein a human helper comes to the 
system's rescue ^K^ienever the user requests it • 



4) In order to make it practical and feasible to use systems 

\ 

such as ours in operat3:9nal environments, we greatly 

\ ^ 

improved the eTficiency of NL&r-SCHOLAR ; not only is the 



output package 5 times faster, bii\ t1ie .overall efficiency 
twice as-' great. 

By^4^ar^,NUie most significant of these advances^ was the 




design and implementation -of a flexible control ^trqctur^ 
that uses the recently released "spaghetti" LISP. > 




structure "'allows NLS-SCHOL^AR to operate on multiple 
environnrents , making it possible for the various * modules of 
the system (the English front end, the^Quizzer, the Tutor 
Scheduler, NLS, the Task MonTTtor, and the Evaluatpr) to be 
handled like jobs in a time-sharing system. That is, 
F^rocesses request "the floor" as need 'arises, and gain 
access to the process queue with preassignable\priorities . 
As a result of this improvement, the system now^ has the 
capability of back-tracking to abbrndoned ^^^pntexts, of 
handling multiple, tasks , -and of coroutining. 

expanded the tutorials (the Primer) from the 
original three lessons to an introduction plus five lessons. 
The fiatet'ial covers usage of the legal combinatiqns of the 
following ^JLS verbs' and nouns: c 

a) Verbs: Load, Print , Insert , Delete, Create, Update, Jump, 
Substitute , ' Set , Reset , Show , Copy , Move , Transpose , 
Output, Help, and the one-character commands ' . ' , V , 

'\\ and <LF>. • 

b) Nouns : 'Character^ Word , Text, Statement, Branch, Group, 
Plex , File and Rest. 

Nuner^s questions, interspersed throughout the lessons 
and forming q'uizzes at ^ the end, test the students' 
comprehension of the instructional material. Over 100 
sup§/vised tasks and "tutor " demonstrations" support our 

4.3 



claim that our users learn "by doing"! 

We^eveloped a prototype Agenda Language that allows us 
to write these lessons in quasi-English format. (The 
lessons were all prepared using NLS and are in indented 
outline format.) The lessons contain not only tasks, 
demohstpa't ions of actions, question-answering periods, and 
quizzes, ' but also branching and remedial loops. The new 
control structure allows us to design muclFi more flexible 
lessons than before, ones that exhibit truly mixed 
user/system initiative. For example, one of the ways we can 
handle students' mistakes is by means of "scratch actions": 
when a student makes a mistake, the system takes over and 
shows him what would happen if the the mistake were enacted. 
This resembles what a human tutor would do ("Here let me 
show you what would happen if you did what you propose") to 
show the effect of the mistake while at the same time 
protecting the student from the consequences of his actions. 

In parallel with this work, our LISP implementation of 
the NLS BASE subsystem was augmented and updated, so as to 
support all the NLS commands mentioned above. We also sped 
it up considerably by using block compiling techniques. 

i 

/ 

/ 

Considerable work was done also on the English ^ front 
end. In addition to questions, this module now handles all 
inputs from the student , including his answers to the 
"tutor's" questions and his "directions" to the system. The 

14 
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semantic network now contains 330 entries, covering the 

commands and NLS concepts which thfe. simulator can handle and 

which the tutorials describe. The output package (the big 

CPU time gobbler in the previous system) was streamlined and 

speeded up by a factor of 5. In addition, the responses it 
produces are nio)re personal and friendly. 

Finally, in addition to the above, 1) we incorporated 
"stop'' and "continue" facilities, so that users could 
proceed with the lessons at their own pace, 2) we began to 
provide users with some feedback on what went wrong when a 
question could not be answered by the system and, more 
importantly, 3) we offered students the help of a human user 
if, they so required (the system looks for one of us, 
establishes a TENEX link, and allows us to come to the 
student's help and to the system's rescue). Contextual 
information (what the student has been up to) is preserved 
in lisp's history list amd is available to us. 

In short, we brought up a new NLS-SCHOLAR system that 
is very much better than the old one in terras of 
flexibility, modularity, capability, and efficiency. 

In the remainder of this section we describe in detail 
the work performed in many of the areas alluded to" above: 
the system's control structure, the tutorial material, the 
English front end, the human engineering features, and 
overall efficiency. t 
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The Control Stru„ci:.up-e - — — — 

The new control structure was designed with s^^eral 
goals in mind: 

1) increasing the modularity of the system to make it more 
understandable and easily modifiable 

' 2) facilita.ting interactions by a) making the "English 

/ 

understanding" portion of tfie system (ENGLISHEXEC) - 
avai/able %t any time by a simple interrupt mechanism, 
anx3 b) allowing the user to experiment with NLS at any . 
.irae without destroying context 

\ / 

3) extending the cai)abil^iti^^s of the ' tutorial material to 



permit branching .-^'and the conditional execution of 
arbitrary INTERLINE functions to perform needed actions. 

- '/ 

The basic id^a underlying the control stfticture is 
simple. The system continually evaluates the priorities of 

/ ( ^ 

several altei^native ggals, .which include ones specifi^ed by 
the user^^^aj)^^^^ by the author of the tutorial 

material . ^^oals with lower pri.ority are postponed, an<J ^he 
highest priority goal executed. Some goals, such as 

"presenting all the tytiorial mat^rj^jal^-inr^ a useful order", 
a^e complex and may continue over a long period of time. To 
facilitate the description of complex, long lasting goals, 
^aW goal is represented by a "process", a collection of 
INTERLISP procedures which when executed will achieve the 

16 \ 
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goal . V 

4 

% 

Because the spaghetti-stack control structure of 
INTERLISP permits any* process to be interrupted at an 
arbitrary point withbut;^ losing the context of the 
computation, complex goals can be reRresented by processes 
which work through a set of sub-goals from beginning to end 
without interruption • A process representing such an 
extended goal may be interrupted and .^ten^orarily, suspended 
to allow other goals to be met. This permits the overall 
system to "stop in its tracks" and interact with the student 
when the student wants help, not just when the system 
decides to pay attention. In this way the control structure 
make? it possible for ,us to design a truly 
"mixed-initiative" system, rather than representing a 
single-minded tutor, since the various goal^ of the tutor 
may be easily interrupted and suspended to allow the student 
to request actions, ask questions, and experiment with NLS. 

The overall control of the system is based in a simple 
"monitor" which acts much like a time-sharing monitor - it 
has a set of suspended processes representing pending 
priorities which must be evaluated, and it chooses the 
highest priority process and permits it to run. 

At any time there may be several pendinR goals, in the 
system, represented by suspended processes. These goals are 
chosen from the set: 

17 
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\ 

a) listening for user commands, questions and answ^fei-Ain 



the user 



English (ENGLiSHEXEC) "■4 

b) deciding what tutorial material to g^ive next i 

c) presenting a tutorial unit 

d) presenting a question ' ^ 

e) waiting to evaluate the answer of a previous question \ 

f) running a student through an NLS task , ^ 

g) providing an experimental NLS environment requested by 

The priority evaluation is implemented primaf^JLLy^^ by a 
stack , but it is made potentially general by having the 
monitor evaluate""a priority setting process associated with 
each * runnable process, and using that to modify priorities • 

k 

In addition/ the stack of processes is easily accessible to 
' running processes, and thus processes can (^nd do) add and 
delete processes on the stack. 

In addition, by making use of' the user-defined 
interrupt character facility and the features available in 
the new "spaghetti sta^^I^ version' of INTERLISP, it is 
possible for the user to internet any process, save. its 
context completely, and start up a copy of the ENGLISHEXEC 
which can answer generkl NLS factual questions, or start up 
\ a safe NLS environment on which 'to' experiment without 

1 

affecting the current NLS environment This enables 
.students to try out risky procedures without fearing the 



consequences of potentially costly mis takes . 

The spaghetti^^ack features permit the entire context 
and state ofXcoraplicated (perhaps recursive) process to be 
sav,ed, ta be run later or exa^nined txy^ other programs , This 
has ^l^een used to implement a "coroutine package" \ijhich 
gf^atly facilitated- Siting simple, easy to understand 
^mod^o-l e^g • 

. 2 Ah example of this is the -^"question posing ^^and 
evafualjion module".^ This module is run having as argunielits 
a question to be posed to 'the student, and evaluation 
procedures for passible answers. It would be easy to write 

a question and to 
interpret t^ next Student input as an answer. However, we 
wished to allow the^^^tud^^ interact with the EHGLISHEXEC 
once the ^question is posed, by asking questions or typing in 
commands i|.f he desires .""^^^^^^^ evaluation must be held 

in abeyaxile until the student actually types in an answer. 

With the coroutine package this is simple - the 
question-posing module calls a coroutine which puts the 
question-posing-raodule on the stack with the evaluation 
section to be run next, puts an ENGLISHEXEC process on the 
top of the stack, aj^d then cedes control to "ttre-^-^nonitor . 




When the ENGLISHEXEC recognizes an input ^aj,..^ an^sw^ 
removes itsplf from the stack and calls the question-posi^ig 
module as a coroutine. To the question-posing module the 

1 
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net result is that the student's answer is made available as 

/ 

if from a subroutine. While this could have been done with 
subroutines, the coroutine technique substantially 
simplifies the state of the' system during the period after 
the question is ^osed.' 

> ■ • i ! 

Tutorial Material , 

* The tutorial material has been expanded considerably 
since November, 1974, and now consists _QfM^i^^^ 
rather than three. These lessons describe the^^^^^-RftSE 
subsystem of the teletype-oriented* version of NLS as it 
appeared in March, 1975; they are written specifically for 
naive users with no previous knowledge of NLS and (perhaps) 
no previous acquaintance with terminals or computer systems. 

New Text - To facilitate the initiation of these naive users 
into the mysteries of computer-assisted instruction, an 
interactive introduction has been written which gives a 
brief description of the goals of the system and explains 

the use of <CR> to terminate commands, <CTRL-A> and <CTRL-X> 

1 

for line editing, <CTRL-T> tb determine the state of the 

job, and <CTRL-H> to get the attention of the "tutor" 

*We use the term teletype to denote generically a hard-copy 
terminal, as opposed to a display terminal. 
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(ENGLISHEXEC) . Thi>^ in troduction supplants and surpasses 
the instruction sheet handout which was used for this 
purpose previously. 

The five lessons differ in both content and structure 
from their predecessors. Revision of the content of the 
original three chapters was made necessary by changes—in the 
NLS , syntax and in * a few NLS commands. The material was 
extended to provide^-faore examples and to present commands 
not previously covered. these new commands include Prints 
File, Print Rest, print the context of the CM (/), Reset 
Viewspecs, and Output Sequential File for producing a text 
file which can be listed on a line printer. A brief 



description of the Help comnand is given at the end^^of the 
last lesson so that the "graduating" student will know how 
to make use of this" facility when he uses NLS without 
tutorial supervision. A small,* self-contained help data 
file about viewspecs has been provided for practice-^with the 



use of this 



command . 



Branching - These changes in content, however, are of much 
less significance than the increased freedom granted to the 
student by the new control structure, and to the author , of 
the tutorial material by the introduction of branches* and 
remedial loops. The ability to U3e branches means that the 
order ^nd the oonfen^y^'^^ is being presented to the 

§tudent can be made dependent on his choices or on his 

17 



performance. The addition of these facilities transformed 
the tas^ of providing the tutorial material from writing a 
text (the Primer) to designi?!^ a programmed instruction 
course.* 





From the students point of view, each lessofrSajQd^ the 
introduction) is composed, as before, of short sections of 
text which are printed at the terminal • At the end of such 
presentations, the student is given the opportunity to 
request mc^e text, to ask any number of questions, or to1 
practice wi^th NLS using any commands that he chooses. 

Tasks - Some^, text sections are followed by tasks which the 
student is a4j<ed to perform. In the course of doing a task, 
the student m^y use <CTRL-H> to get the "tutor's" attention; 
he nay then ask questions, practice with NLS to see the 
effect of a command ,~~ask that he be allowed to restart the 
task, or ask that the task be done for him. If the student 
performs the task, his work is evaluated and help/ul comment 
or criticism is provided. If his work is unsatisfactory he 
y be asPred to do the task again, either wholly or 
partially. 

Questions - Some text sections are followed by questions for 
he student ~^--tQj answer. In the course of^' trying to answer 




question he may ask'-<5lre^2Lti^^ himself, or practice with 
Ntrs in an attempt to determine thV apswer. 

22 . 
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A set of questions (a quiz) has been placed at the end 
tl3^--^4^^ two lessons so that the 

student ma^^^^^TTB-v^i^^t^ additional method of assessing his 
progress. Answer*'5=::,...,gre^ and appropriate responses 

made. Considerable latitude is pT^ided in the judging of 
answers so that the student is not constr^ained to . a 
particular form. For example, the question "What is the 
statement number of the origin statement" may be answered 0, 
statement 0, or zero; all are equally correct. In cases in 
•S . which an answer has several parts, missing information is 
often supplied in the evaluation. 
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Answers - The handling of students' answers is made easy by 
the use of answer predicates. A sequence of these 
predicates can be written by the author after each question; 
the predicates are then tested one aft'er the other until one 
of them succeeds. They operate in two steps: the first one 
provides for extracting expected words, for testi^ t^se 
words in various ways, and for filtering out irrelevant 
parts of the answer; the second step is some action which is 
undertaken or not, depending on the outcome of the first. 
These actions generally consist of some tgxt being printed 
followed by an optional branching instruction. 

English Front End 
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The English front end handles all l^rlguage input from the 
.^tudent. It t^kierefore must/ be powerful enough to 



distijiguisVi between commands, /Hj^^kta lesson " 

^^^^^^ ^^^""^ — ^ ' iTn 

2"); queries, ("How do I priniKthe whole file?' 



Delete 



replies to tut'&r^gene.^^ted questions, ("The ^atemehts are 
and 4B"). We d^ided^ to use tjie. j^j^o-tion" <5T^^dfeantic 




grammar [Burto^)Xl975 ] with two imp^^r<an^ additions, namely 
Instantiat i^fT of variables and Case ass^bgnments [Fillmore 
1968]., 'These two processes will be described lat>^. 

The key notion underlying the semantic grammar apprbach 
to parsing is the replacement of the search for syntactic 
constructs by a search for semantic pnes. Parsing a" 
student 's request in this way yields -its meaning directly , 
i.e., it produces an executable retrieval formula /that 
prescribes a search in the syst^em's "data base" (the 
semantic network plus the user's work space). The search 
can then be carried out and the results used to synthesize 
an answer to the request. Notice that in such a- parsing 
process there are no separate syntactic and semantic phases 
(as there are in systems like the LUNAR parser [Woods 
^ 1972]) . 





The Parsing Process- The parsing process begins with a 
prescan of the student's input. Abbreviations are expanded, 
synonyms are recognized and rewritten into a canonical form, 
and compounds are collected into one word. These processes 
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ease the work of the parser itself by cutting down on the 
nuraber of alternatives that must be considered/ 



After the input is prescanned, an attempt is made to 
pars>^it using an embodiment of the grammar described in BNF 
in Figure 1 . Each non-terminal node of the • grammar is a 
semantic category which takes into account alljthie predicted 
ways of expressing it. Each semantic category is. embodied* 
in a LISP function that tests the input /string (or a ' 
substring of it) to determine if it belongs to the category. 
If successful, the function returns a value which condenses 
the "meaning" of the string. 

e top level rule is <REQUE,ST>, which can be realized 
by Vour ° semantic categories: <DIRECTIVE>, <QUESTION>, 
<NL^CTiqN/REQ>, and <ANSWER> . This means that an input 
from tn^\user Ca request) can be either a directive, a 
question j^'ahN^fc^^ expressed in English, or an answer 

to a question ask^id previousj^ by the system. Each 
alternative is tried sequ><^ially until one succeeds. If 
none succeed, an error message is typed to the student ("I 
didn't understand that. Please rephrase.") A good way to 
describe the parsing process is by example. We shall follow 
th^ parsing of the request "What command prints the next 
statement?" (^e Figure 1). 
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<REQUEST>:= <DIRECTIVE> 
<QUESTION> 
<NLS//^CTION/REQ> 
<ANSWER> — 

<DIRECTIVE>:= ? ! CHECK ! PLAY ! RESTAf^T ! GO ! HELP'! STOP 

<QUE^TION>:= <DEFINE/REQ> 

<WHATIS/REQ> ' 

<CONTENT/REQ> 

<PARTS-IN-PART/REQ> 

<PARTS-IN-LEVEL/REQ> ' - 

-jfROCEDURE/REQi 

<TYPE/REQ> • 

<INSTR/REQ> \ V 
<POSITION/REQ> , . ^ 



<NLS/ACT-ION/REQ> := <ACTION/SPEC> \ 



<ANSWER>:= <THE-ANSW&R> ' ^ ^ ^jr^ 



<D0NT-KNOW-ANSWER> 
<LIST-ANSWER> • 



<THE-ANSWER> := t THE THEY IT] [IS ARE] 

<DONT-KNOW-ANSWER>:= TELLNME ! I DON'T t(Nt5W 

<LIST-ANSWER> := a list that doesn't begin witk a <VERB> 

or a question word like What^,.I^, Why, etc. 

<'dEFINE/REQ> := [DEFINE DESCRIBE] <NOUN> 

WHAT DOES <NOUN> [DO MEAN STANDNFOR] ^ 
HOW DOES <NOUN> WORK 

<WHATIS/REQ>:= WHATMS* 

'[PURPOSENOF <NOaJN> 
CONTENTNOF <STR+ADDR> 
LEVELNOF <STR+ADDR> 
PROCEDURENFOR' <ACTION/SPEC> 
ADDRESSNOF <STR+ADDR> 
EXAMPLESNOF' <NOUN> • 
EXAMPLENOF <NOUN> 

DEFINITIONNOF <NOUN> ■ ^ 

■<CURRENT/PAR1>«> 

<STR+AbDR> ^' 
<NOUN>] ■ 1 
*Also SHOWNME TELLNME GIVE\ME TELL\ME\ ABOUT 
WHATNARE 

<CONTENT/REQ> := WHAT <STRUCTURAL> CONTAINS <STRING> 



Figure 1. BNF description of the grammar. 
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<parts-ii^-part/req5|Pwhat <structural> are in <F1LE/PART> 
what are <structural> in <file/part> 

<PARTS-IN-LEVEL/REQ>:= WHAT <STRU.CTURAL> ARE <LEVEL/PART> 

<PROCEDURE/REQ>:= [HOW\DO\I SHOW\ME\HOW\TO TELL\ME\ ABOUT] <ACTION/SPEC> 

<TYPE/j?EQ>:= WHAT CAN I TYPE AFTER [<VERB> <STRIMG> <PROMPT>] 
WHAT CAN FOLLOW [<VERB> ^STRINO <P^ROMPT>] 

<INSTR/REQ> := WHAT (COMMAND) <ACTION/SPEC> 

<POSITION/REQ>:= WHERE AM I / 

<WHERE IS/ARE <STR+ADDR> 
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<ACTrON/SPEC>:= <VERB> [<OBJ>] 

<VERB>:=, word whose part of speech is Verb 

<OBJ>:= [<RELATIONAL>] [ <NOUN/PHRASE> ] [<OBJ>] 

<RELATIONAL> := words like NEXTXTO FROM AT TO, etc. 

<NOUN/PHRASE>:= <TASK> 

<STR+ADDR> 

<FILE> 

<NOUN> 

<TASK>:= TASK <NUMBER> 9 

<STR+ADDR>:= <FILE/PART> 

THE <STRUCTURAL> <STRINp> 
THE <TEXTUAL> <STRING> 
<CURRENT/PART> 
•<STRING>' 

<FILE>:= (NLSNFILE) [BREAKFAST DINNER MYBREAKFAST] 

<NOUN>:;? any word whose part of speech is Noun' 1 

<NUMBER> := a number 

<FILE/PART>':= STAT^MENTNO 

. /GROUP <ADDRESS> <ADDRESS> 

[STATEMENT STATEMENTNNUMBER. BRANCH PLEX] <ADDRESS> 

<^ADDRESS>:= a word whose* first character is a number 

<STRUC;ijRAL> := STATEMENT ! BRANCH ! GROUP ! PLEX 

<STRING>:3^a string delineated by double quotes 

Figure 1 (cent) 

{ 




<TEXTUAL>:= WORD ! CHARACTER ! TEXT 



<CURRENT/PART> : = CURRENT\WLS\COMMAND • 

CURRENTWIEWSPECS 
CURRENTNSTATEMENT 
\ NEXTNSTATEMENT 
BACKNSTATEMENT 
CURRENTNADDRESS 
POSITION\OF\THE\CM 
CURRENT\STATEMENT\NUMBER 
CURRENTNFILE 



Figure 1 (cont) 
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In the prescan the words "next _ statement" are 
recognized as a compound word or concept and are rewritten 
as nextXstatement. Starting with the grammar rule 
<REQUEST>, the first check is to see if the sentence is a 

s and the next one is tried. 
<QUESTION>. The first seven realizations of the rule fail; 
but <INSTR/REQ> succeeds with "What" being followed 
optionally by the word "command", followed by an 
<ACTION/SPEC> . <ACTION/SPEC> succeeds, since "print the 
next\statement" is indeed an action specification. 
<ACTION/SPEC> returns as its value (remember it is a LISP 
function) an expression that is the "meaning" of the action 
specification: , • 

((VRB PRINT) (OBJ NEXTXSTATEMENT) ) 

This says that the action is represented by the verb 'Sprint" 

and 'the object of the action is "next\statement" . in turn. 
If. ' • 

<INSTR/REQ> returns: 

(QFIND/INSTR- ( (VRB PRINT) (OBJ NEXTVSTATEMENT) ) ) 

which represents the "meaning" of the' sentence. At this 
point the parsing phase is complete. 

To find the correct answer,. this "meaning" is executed 
as a LISP expression. (QFIND/INSTR is the function and VRB 
and OBJ are its arguments)-. The function QFIND/INSTR first 
checks to see if there is an OBJ. If there is one , • It looks 
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under the OBJ's data base entry for a section of data base 
beginning with the VRB. If that search fails, a general 
reply is given by finding alj instruments (commands) under 
the VRB print and printing out the procedure for using each 
one. In this way, most of the knowledge the data base 
contains about printing would be given to the student. The 
belief is that a complete description is better than a 
simple "I don't know". Among all these procedures, the 
student may find the one he was looking for. 

In our example, the search for the VRB uiider the OBJ 
succeeds (see figure 2). 

Figure 2 



NEXTXSTATEMENT 

(PRINT (I 2) (AGENT NIL USER) 

(OBJ NIL NEXTXSTATEMENT) 
(INSTR NIL <LF>\COMMAND)) 



The English output routines take the piece of ^data base and 
form the English sentence: 

you PRINT THE NEXT STATEMENT USING THE <LF> COMMAND. 

i' 

Fuzziness - The parser allows for fu4;ziness; that is, it is 
able to skip over words in a/controlled way in order to 
achieve a parse. The hop$ is t^t these words are noise 
words or at least that the^ can be ^skipped over and still 
permit a parse tha^t is not/far from the real meaning of the 
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request. The problem is that in some cases fuzziness leads 
to a completely different meaning. For example, consider 
the sentence "What are the default views'pecs?" . In pushing 
for an object^ let's say the parser doesn't recognize the 
word "default". Fuzziness would" allow the parser to skip 
over this word.' It recognizes "viewspecs", and in effect 
parses the sentence as "What are the viewspecs?". Applying 
fuzziness techniques well is a very tricky business! 

Instantiation of Variables - An effort was made to see what 
it would take to build an English front end for NLS that 
would allow the student to express NLS commands in English. 
The added bonus from this research was the ability to answer 
with greater precision questions that de^alt with more 
specific information than the data base explicitly contains. 
An example is the sentence "How do I delete a structure 
unit" versus the more specific request "How do I delete plex 
2?" This ability was achieved by adding to the data base a 
new construct: instantiation variables that may get set 
during parsing and, if so, will be used in place of the 
general term — otherwise the more general term is used . 
For example, in the data base entry for DELETEXCOMMAND , the 
string $INS appears 3 times*. Each time it is followed by a 
variable name, (XOBJ, XOBJSTR, or XADDSTR) and then followed 
by a regular piece of SCHOLAR data base (see Figure 3). 
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Figure 3 



DELETENCOMMAND 

[PURPOSE (I 2) (DELETE NIL 
(AGENT NIL USER) - 

(OBJ NIL ($INS XOBJ ($EOR (NAME NIL (OF NIL STRUCTURENUNIT) ) 

(NAME NIL (OF NIL STRING\UNIT) ) ) ) ) 

(INSTR NIL DELETENCOMMAND) 
(PROCEDURE NIL (TYPE NIL 
(AGENT NIL USER) 
(OBJ NIL ($SEQ "DELETE " 

[$INS XOBJSTR 

($EOR (NAME NIL (OF NIL* STRUCTURE\UNIT) ) 
(NAp NIL (OF NIL STRING\UNIT] 
(iJNS XADDSTp ADDRESS) 
<XR> <CR>] 



In processing "How do I delete a structure unit" none of the 
instantiation variables is set and so a general response is 
given: 



YOU DELETE A STRUCTUR-E UNIT OR A STRING UNIT USIN-G THE 
DELETE COMMAND. 

PROCEDURE: YOU TYPE 'DELETE FOLLOWED BY THE NAME OF A 
STRUCTURE -UNIT OR THE NAME OF A STRING UNIT, THE ADDRESS, 
<CR>, AND <CR>. 



In processing "How do I delete plex 2", all ' of the variables 
are set during parsing so a very specific reply can be 
given: 

YOU DELETE PLEX 2 USING THE DELETE COMMAND. ^ V 

PROCEDURE: YOU TYPE 'DELETE ' FOLLOWED *BY • ' PlEX ', '2', 

<CR>, AND <CR>. 



Now, not only can the question be answered , but it can 
be turned into a command to NLS to perform the action 
"Delete plex 2" on a copy of the user's file. It parses as 
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an '<NLS/ACTION/REQ> . Thor Torm returned from the parse is 

(QDO/ PROCEDURE (VRB DELETE) 

(OBJ PLEX (ADDR 2)) ^ ^ ' 

\ 

QDO/PROCEDURE- is a function which first retrieves the^ 
appropriate piece of data base and checks to see if all the 
instantiation variables in this piece are Tilled in.' It 
then calls LISP-NLS, handing dov/n to^t^Jjl}^ legal command 
sequence. (If all the instantiation varlfables were not set 
during the parse, a reply is generateT'TeTli^ig"^^ 
what is missing.) Using a copy of the student's current 
file, LISP-NLS executes the command sequence: 



BASE C: Delete C: Plex (at) A: 2; 
OK: ; 



Further uses of LISP-NLS to answer questions - We have just 
described"*^ne use of LISP-NLS: responding to an English 
request to^have" NLS pert^or^TT^ i^command . A second use is to 
respond to queries like "HH^re am I now" and "What is the 
address of the statement^cojaJLai^^ These kinds of 

requests imply that at least one NLS command be performed. 
In the first case the answer can be found by performing the 
command; in the second by performing a series of 
commands - Jump Address 0, Jump Address "PRIME", then "." to^ 
get the current address. 
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Human Engineering 





ler to najce NLS-iSCHOLAR anj easy Ind^ pleasant 
system to^'lr&e^^we s t r i ved^lio^n d ow it with a number- of^Bttman 

a>^ill be described nexJ;. 

Stop ^ '^nd resume > - Sessions with NLS^-SPKSLaR have natural 



breaking points, such as lesson boundaries -or large topic 
changes, at which it is convenient and even desirable for a 
user to quit. Having stopped at ■ one of these 
system-provided breaks, the user can resume the lesson at a 
later time by asl<ing the system something liice^"'"'**^4iart lesson 
^ ,now, pjease". Often, however , users find the time bet>e€Ln 
these /natural ^brealcs '^rr^e-^^oi^ either because their 

own performance has req/iired^x-s^longer time than average, or 

because something ,.a4yfp — dem^.ds JJiglr attention. We have 

provided the system/ with the necessary mechanism for 
allowing those users to stop- the lesson at any time, in 
whatever situation /they may find themselves: in the middle 
of a lesson, performing a task, answering a question, or 
even working witn NLS doing their own thing. All they have 
to do is get/ the attention- of the "tutor" (by typing 
<CTRL-H>) and £hen tell it they want to stop. The system 
responds ' by^/ asking the user to confirm his" request and to 
indicate if/he intends to continue at a later time. If both 
answers are affirmative, the system writes out a file (a 
LISP SYSOUT file) in the user's directory. When the user 



34 



30 



comes back, the system remiads him of the existence of a 
suspended work session; if the user .waats to, he can 
continue exactly where he left off by simply typing RESUMfi 
(which causes a LISP SYSIN) . This feature was very sorely 
n^ded and was used by almost all those involved in the 
field testing. 

- Getting help from an expert . - Since we did not expect our 
system to be able to comprehend all user requests and to 
always provide useful answers, we endowed NLS-SCHOLAR with a 
feature that allows a human expert Ao come smoothly to the 
System's rescue when^ the system fails. This facility 
operates as follows. Let's suppose that a user is in the 
middle of a task, asks a question whose answer is badly 
needed, and the system either fails to understand his 
"'^""^cjuestion or gives him an unsatisfactory answer . ^ If he asks 
for help at this point, the system will seek a logged-in 
human expert, establish a link, and report the failure to 
• the expert. If it isn't posrsible for the expert to provide 
the answer solely on the basis of an isolated question, he 
can examine a history list maintained by the system. This 
list is a record of previous interactions between user and 
system which provides the context the expert o'ften needs to 
an.aw;er^a question appropriately. 



The main reason for the incorporation of this facility 

was to allow our . students to utilize lesson time more 
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effectively; we wanted tlieir experience using NLS-SCHOLAR to 
be a profitable one in spite of the system's limitations, 
and we hoped the facility would minimize frustration and 
unnecessary breaks • In spite of our hopes, the facility was 
hardly used at all: only one of our users ever attempted to 
take advantage of it, but unfortunately no expert was 
logged~in at the time help was sought. 

Question mark. - Given the great flexibility of the control 
structure, the student nay well be confused as to what to do 
when he gets the ^'tutor's" attention. A question mark 
facility was implemented to help users^ remember what they 
could request the system to do for then. When the student 
types a the system responds with a list of one-word 

commapds which may be used to initiate actions, such as 
starting a lesson, restarting a task, stopping a lesson, 
resuming it, summoning help, calling NLS, etc. These 
actions are not necessarily invoked specifically by their 
asso^ciated command; rather, it is the combination of command 
and situation that decides which action will be undertaken. 
Thus, if a user types "continue", several ^ things may happenr 
a) if he ^was in the middle of a lesson, the lesson 
continues; b) if he was performing a task, he goes back to 
the task's environment; c) if he just entered NLS-SCHOLAR 
a-nd tkigre is a stopped lesson under his name, the lesson is 
resumel]^;^ d) if he was working with NLS doing his own thing, 
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he is returned where he left off. 

Efficiency - The newly brought up NLS-SCHOLAR system is 
remarkably more efficient, in terms of CPU utilization, than 
its predecessor: it takes about 3 minutes of CPU time, on 
the average, per lesson hour. This efficiency measure 
applies to a lightdy loaded TENEX system; under these 
circumstances the lesson proceeds at a good fast clip. 

This relatively good efficiency is due to three 
^^^provements made to NLS-SCHOLAR. The first improvement was^ 
to redesign and streamline the output routines, the ones 
which are responsible for producing English sentences out of 
information encoded in the semantic network. This, resulted 
in a ^package that operates 5 times as fast as the old one. 

The second improvement was to block-compile LISP-NLS. 
This technique provides a way of compiling Several functions 
(LISP routines) into an entity called a block. Once a block 
is entered, function calls within it are very fast and 
variables' values are looked up directly, resulting"^ in 
considerable execution speecf-ups. It is not rare to see 
order of magnitude improvements from judicious use of this 
technique. ^ 

The ^hird big Ijiiprovement was to pre-compute the tasks' 
vectors. Previously, when a user's performance of a task 
was to be evaluated, the system used LISP-NLS to periderm the 
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correct sequence of commands and to obtain the correct image 
of the work space. This was then^ compared with the result 
of .the user's commands, Jn the present version of 
NLS-SCHOLAR, these correct images are obtained for each ta^k 
at system generation time, and are stored away in a separate 



file- 



A filfe^andle is provided for each task., and is- made 
.accessible from the semantic network entry for the task so 
that the correct image can be retrieved from the file. 
Consequently, when a task is evaluated there is no CPU time 
wasted in generating the correct image. 



SECTION III OPERATIONAL TESTING AND RESULTS 



As described at the beginning of Section II, - 
"operational" testing of successive versions of NLS-SCHOLAR 
started early in the course of our work. For this purpose 
we used BBN personnel ranging from completely naive users, 
through secretaries with experience using other 
computer-based text editors, up to experienced computer and 
behavioral s'^cientists • 

When our system was (reluctantly) pronounced ready, it 
was used in an informal but realistic testing environment by 
14 non-BBN users • Among them were DOD personnel from the 
Air Force Data Services Center — an outfit chosen by the 
Contracting Agency — whose sophistication in using NLS 
ranged from very naive to experienced. In addition, the 
Contracting Agency solicited an indepemient evaluation from 
qualified Technical Personnel of the Information Sciences 
Institute (ISI) of the University of Southern California, 
The results of this evaluation are described in a report 
which is included in this document as an Appendix. 

The data obtained from the operational testing is in 
the form of dribble protocols recording ^he "dialogue" 
between users and NLS-SCHOLA^. Over 50 protocols were of^ 
significant length (ranging from 20 l-o 90 minutes of on-line 
time) to be considered useful and t>o.^wairi=int their analysis. 
In ^ addition totWL^dat'a, an amount royghly equivalent was 



obtained via our own internal testing using BBN's personnel. 
Taking everything into / account , protocols representing 
approximately 100 hours of on-line time were .analyzed . This 
amount 9f data is not sufficient to' establish statistically 
valid results, but it is Enough to sustain very defin^ite 
qualitative conclusions about the system's capabilities and 
limitations. 

General Results 

^ The main thrust of this section is to describe and 
discuss a number of specific problems and problem areas 
identified in the course of the field testing* In order to 
frame the descriptions and to focus the discussions, we find 
it necessary, at the risk of being considered unsoholarly, 
to present the general results of our analysis here rather 
than at the end of this section* They are: 

1) The tutorial set-up appears to be very effective. New 
information is presented in bits and pieces of digestible 
size and users are kept on their toes (albeit in a very 
friendly environrSent: ) with dozens of questions they are 
asked to answer and NLS tasks they are asked to perform. 
Users do learn NLS: this is ^ evident not only in the 
progress of their work , but also f'pdm personal 
communications (telephone calls , messages , and link ups). 
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2) Thf •'suprervised task environraen t»» , whereby the system 
evaluates the results of a usejr^s performance of an NLS 
task and offers comments about it, appears to be, very 
valuable. The system succeeds in pointing out mist^4^ 
and provides information useful for rectifying them. 



However , the system is somet imes over-zealous (rejecting 
outrightly th.e performance of a task for so^e trivial 
discrepancy) and sometimes fails to point out some 



erroneous action ^ underta^e^n by the user. These 
shor^tcomings ' are not / sei^ious but they detract from the 
,system's |^intelligen€" ^a^earance . 



/ 

3) A substantial ;par)fe^ of) the system's "smarts;* resides in 
its English fronV end; NLS-SCHOLAR is^^^esigned so that 
the usej^can the initiative anytime it is his turn 

to type and formulate requests (usually questions) to the 
system. Not surprisingly, hoover, this feature of 
NLS-SCHOLAR performed less satisfactorily than the re^t 
of the system; only about 1/3 of the requests formulated 
were ^ answered relevantly and usefully. This poor 
performance may have^ inhibited many users from asking 
more. quest ions .// - 

.In viev of the results outlined above, the rest of 

' / / . ^ ^ 

'^his section is concentrated on a detailed discussion of 

/ the {performance^ of our English •fro^t end, and on the 

ger^eral ^issues it raises in the area of Natural Language 
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Comprehension ► 




- 'V. 



Two points must be considerfe<J_Jyi order to view this 
last result in the proper perspective. In the first 
place, a large majority of the requests that the system 
failed' to answer or answered incorrectly could have been 
handled satisfactorily with minor changes to the \ system 



and additions 



its semantic network . 



Undetected 



spelling errors, unanticipated synonyms, common but m 
anticipated sentence syntax, lack specific ktiow ledge , 
etc, are examples of problpflTs of this loLnd which af^e 
relatively easy to-'-;;t^ti fy as each one is round. As a 
whole, however, ^uc>n time and effort must be expep^d^^^^^^ 
eradicate such prob\gm5'''''9ntirely . 




Secondly it mug^.* ^J>^<borne in mind that the tutorial 

mat-erial • is very clear and complete. Vt leaves 

/ ^ • 

relatively little room for doubt *Vithin th^ domain of 

^ / 

procedural and conceptuayf^knowledge /^that the question 

\ ^ ' • ' 

answering ^ystem is deaaf^ned to handle. Consequently, 

the relatively few yinanswered reciuest3 not covered in the 

"easy problems" 

combination 





sc/ib,ed above, reflecte<I^ a 



subtler /doubts aftd the efforts 



sophisticated user^ to p^ncoct a question to assess /'the 
system '3 .^pabilit 




These questions remained unanswered either because 
they were expressed in round about ways (i.e.', outside 
the s*et of paraphrases the system can recognize^'^^'or'^^ 
convoluted sentence structures), or because they were 
iiW^c^sely formulated. The round about problem was not 
important in our case. It is 'more likely to occur in 
questions' posed by users returning to the system after a 
partially forgotten previous exposure to its tutorial % 
material. This situation could not deV^elop with,in the 
period the system was tested. 

, \ ■ 

Imprecisely ^£ormulated requests' were much more 

\ A.. 

common, within the .relatively small number* of 
hard-to-answer questions' we are focussing on, than 
precise circumlocutionss . The^ relatively high frequency 
, of imprecisely formulated requests antt ^heir i^ir^ent 
interest justifies the more detailed desgi 
analysis of their nature which will be fp^jad--''T:ate^ in 
this section i 





"What is my current staten]ent>?" 
or "What does OK/C mean?" 

In the -first case, the system's spelling error correction 




list jpontained both the words "statement" and 
"statements", which resulted in "stateinent>" '-^being 
corrected to "statements". The system knows what a 
"current statement" is (both, the meaning of the concept 
and how to find out .its present^ value), but it was 
hopelessly confused by "cur^^ Given our 

current approach that^ emphasizes speed and expediency, 
the remedy is to eliminate "statements" from the spelling 
correction list. Ai' better solution, such as performing 
morphological analysis and checking the agreement of verb 
and predicate numbers, would .have required a 
fundamentally different approach, 

-In the second example, the system knows the meaning 
of most prompt symbols', and in particular that of the 
/C: prompt' (notice the colon). While the^ "system i3 
^prepared to accept many common abbreviations and 
misspellings of these symb.ols, Oi^/C was no.t anticipated. 

Unanticipated synonyms A very common group., exemplified 
by, - • 

"Please review the one-character commands" 

•"How do I logout?" ' ' ^ . . 
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"Explain -the OK: promp^t'^ 

The system would have answered these requests correctly 
if they had contained the verbs "list" or "tell me about" 
or "give me " instead of ".review" ; "stop" or "quit" 
instead of^ "logout"; and "describe" instead of "explain". 
Fixiag-^this may be trivially done by incorporating those 
ve-eh^ itr-*rfe€ internal synonjyif lists- of the system, or by 
incorporating their d^J^llntions and usages in the 
semantic network •.^^-'^serve that "review" could have been 
used to mean something different from "list", ,e.g., to 
mean something like "Let's go j>jijer:::^the^ 
'^<5ommands again", after, they have been presented once. To 
^handle this kind of request is feasible but less trivial. 

Common but unanticipated syntax 
"What NLS commands are "there?" 

The almost equivalent question ("What are the NLS 
commands?") would have been answered correctly, but that 
format ' didn 't occurred to this particular user. (Neither 
did we anticipate the "are there" format!) 

Lack of knowledge - A few examples are, 

"How do I delete a file?" 
^"Tell me about control commands" 
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"What is an illegal delete?-" . 

Deleting files was not possible from within NLS until 
very recently. This negative knowledge should have been 
incorporated in the semantic rietwork, enabling the system 
to generate its canonical negative answer ("YOU MAY NOT 
DELETE AN MLS FILE USING AN NLS COMMAND"). This 
particular question generated a rather amusing dialogue 
(albeit very frustrating for the user) in which the 
system told him how-to delete all sorts of things, except 
^files. Also, although grouping certain commands under 
the category of "control commands" is certainly 
reasonable, the category was not anticipated and an entry 
for it is" not contatined in the senantic network. The 
same applies to "illegal delete"* 

Pdor answers. - In some, cases the answers generated by 
.the .question answering system were unsatisfactory. 
Examples are: 
^'Describe play"# 

PLAY IS' THE COMMAND YOU TYPE TO THE TUTOR . TO .PLAY WITH 
NLS. 

"What do you mean play with NLS?" • ^ • 

PLAY IS THE COMMAND YOU -.r- " 

"What is a field?" 

A FIELD IS A SEQUENCE OF CONTIGUOUS DIGITS OR- CONTIGUOUS 
LETTERS WITHIN A STATEMENT NUMBER. ~- " 
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"What does that mean?" 
I?LEASE .REPHRASE YOUR REQUEST. 



Obviously, giving more coherent and complete responses is 
. something that can be taken care of. In the above case, 
since "field" is' a particularly difficult entity to 
define in a more useful way, a specific example c"buld be 
appended to the definition (FOR EXAMPLE, THE FIELDS ^ 
12A2 ARE 12, A, AND, 2). 

Unanticipated environments > - Many of our users worked 
on display terminals. Sinc^ NLS-SCHOLAR' is geared for 
users of hard copy terminals, many of the requests 
originating in^^display environments were not contemplated 
in our design, .For example, users of display terminals 
. would often forget what task they were supposed to 
perform, or what question they were expected to answer. 
Every time the system reminded these users about their 
pending' obligation , they would request a restatement of 
the question or*task. Handling such requests ("What was 
the task?", "Ask the question again", ^^artr^di^''Vq^ 
me to do?'") is possible, but again, we did not anticipate 
them . ^^^^ 

As can be infe^rred iVom rtr^'^precei^ing descriptions, thB 
common, trait, that unifies these classes of failures is our 
incomplete an'tioipation of what users would do in otherwise 



expeicted situations. An elucidating comment to forestall 
our readers from reaching hasty conclusions is in order 
here. , 

It is very easy to disregard the importance of these 
"little failures" and attribute them solely to our lack of 
foresight. This condescending attitude, that can perhaps be 
subsumed as "How could they have forgotten X, or not taken Y 
into account?"^ fails to perceive the real issue. It is 
false to believe that incorporating X or bringing Y into the 
fold will make a substantiajL difference. The authors of 
this report did nothing elsf during the last 2 months of 
their work, and still the system is plagued witt) "little 
problems"! The crux of the matter, what must 'be recognized, 
is that when one faced with the fantastic 'variety, the 
multitudinous aspects, and th^ changing modalities of the 
behavior of a human engaged in a dialogue with a machine, 
converging to a system relatively free of these "little 
problems" is a very long process. All we can say at this 
time is that this first round of field testing has been 
extremely useful in uncovering a large number of problems of 
tbis typ§, and that we expect the next round to uncover a 
smaller number. 

The Harder Problems 
We turn our attention now to the more interesting 
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failures of our English front end, those involving questions 
that were too imprecisely formulated for our system to 
answer. The imprecision of these questions stemmed frOra**- 
anaphora^hey contained or, more seriously, from their 
"situational" character; tha^ is to say, cc^pr^|hendi^£^^^ 
would have involved understanding the process of the user's 
interactions with the system. These questions aros'e in such 
a form because the user assumed that the system w3s aware of 
the entire situation as it api^ared to him; it is^ surprising 
to see how large an amount of contextual .information, must be 
taken into Recount before such questions can be properly 
iinderstood .* 

The difficulty resides not so much in the literal 
interpretation of questions,^ as if they wjbre precise 
ormulations of the specific bits of knowledge \ the 
questioner might seek, but rather in, figuring out what each 
particular person may** have meant to ask, given his 
background, his previous experience, his previous 
performance, what he ought to know vs. what he seems to 
have learned, the environment he is working on, etc. These 
are very hard problems; they lie at the heart of the- Natural 
Language Comprehension research area and their general 
solution still eludes us . Our purpose is to explain why 
these problems are so difficult, and to-* show the 
advisability of indirect solutions. 
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Many of these problems are rather sub^Xe^-^a^it. is easy 
to dismiss th^m because one '^^^oam — oCien stumble upon a 
seemingly general solution whose regil. underlying "ad-hoc" 
character . is h'ard to perceive. y6 appreciate the 
difficulties involved, we shall see Rcsrv/ a solution tha^ 
seems satisfactory for ^ particular problem fails to apply 
to an apparently similar one, .We shall proceed by analyzing 
five scenarios taken from our, protocols. Each scenario 
comprises a \ description of a particular situation, the 
relevant context, and the question formulated. The 
scenarios ar^er-r^t^^ked ^in order of increasing difficulty, in 
terms„,.....-0^"''^he mechanisms that have to be invoked in order to 
handle thenT. 




Anaphoric reference 



First scenario - The curtain rises after . the stuoSTit has- 
been taught the purpose and usage of a fairTy large number 
of "viewspecs" - characters used to specify how an NLS file 
is .to be printed or viewed. Before leaving the subject > the 
system mentions several additional vi.ewspecs, and then tells 




As you can see, there are a great many viewspecs. If 




JLikely 



purposes 
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At tljia, point ,^ the 7Student asks 




ey control?" 




This'' example 'is deceiving. .'Beoawe it would appear that 
handling such a simple anaphoric reference is within phe 
state of the art [Woo^s 1972]. The difficulty, however, 
j?.e3ides^" in the lack of coupling between the question 
answering system and the tutorial material; in other words, 
the question answerer does not know th^ details of--wtia^tJ^^ 
t finished teaching and cannot pl-acS^^^Trtm^ 



tutor has jus 
request inT^tJontext . 

A conceivable way to aope 'With this probleni;*would be to 

have a complete internal representation (in the semantic 
♦ 

network) of the tutorial materliSrH^^and then synth^siz-e_ ^the 
text the user reads from that internal representation. 
Given the present state of our knowledge on how to represent 
information in a semantic network and how to generate 
passable English from it, such an approach would fall short 
of our needs and would be totally inadequate for teaching 
naive users. 



Another way to cope with this problem would be to 
re-write the text so that such anaphora^^^s^cuJld be inhibited 
from occurring, rather than^ being encouraged as thejT'aT^'e^in 
this example. The studeirt is likely to frame his questions 
in terms of the. words of the text, ("If^you are interestgiL 
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^ 



in what thev cgntrbi'"), so the elimination, of peferential 
pronouns m the text might .encourage him to eliminate 'them 



in his questions. 



But even if we could synthesize thei;text ? gracefully 
from a semantic network or re-w'rite it carefully with an eye 
towards forestalling anaphoric questions , other dif f iculirlTes 



bjjld ariser-^a^ ipdioated by^^^fre-^ext scenarios . 



Elliptic structure , , . 

Secon d scenario - The system tejls the student: 

NLS' FILES 



In order to begin using NLS you will need to sp 
^hich 'f41e' of inf orrna4^ion you want to work with, 

Eacinr f ile is sort of like a notebook "or^^nToTcler 
which you q^zl keep information,^^- ^ ^ 

You may keep as many different notebooks (files) as 
you like. 

Files are automatically^ st<3red_w hen yo u are not using 
them. . "^^^ 

Before you can work with a Tile you must 'load*^ it from 
tfre storage into the working space- of the computer^*- 

^ . - ( 

Each file ha3 a name so you^may refer to it easily* 

File names are made up of letters and digits and may 
be quite long \ like BUDGETFORFISCAL75 • 

No distinction is made within file names between 
upper and lower case letters, - both are treated as 
the same char*acter. 




52 



48 



'What 




character's?" 



iBefore going on to point^ui th^ new prt^fe^eins inherent in 
t*h is example, we shoju Id mention irf^-^p^ssing that, TTen^agairr, 
the occurrence of ^hl s quest ion couJjOiaVe^^ prevented b>r 
re-writihg the text so ,that it specified in exact detaiL 
what .characters could be used in designating file names, 
I^s— would, .provide , however, more detail than most users 
really want and is the sort of information i:hat belongs more 
properly in a reference manual than in a tutorial.) - 



Let us ignore the problem of the conjunctional form of 

entlyunable to handle, and 



blanks?" 




nis^ere are the^"~^lTiTrtl-^ 
noV a sentence) an;^!^ the 



pt^blet/ referrents.^^^ For 



sen ten^re^ littered by th^ 



UPPER CASE 
BLANKS. 

If- instead* one ; 

djie might gene>ate the answer 




SAME' - AS ^LrOWER. .^GASE 



4^han on^ file name, 



YOU MAY KEEP BLANKS (as well as 
characters) IN FI}.ES» 
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In order to generate the answer that the' student is 
actually seeking, i,e,, 




FILE NAMES MAY NOT CONTAIN BLANKS 

a crucially important new component: a model of the 



a model )>?ould be used, perhaps unconsciously, by a 



human tutor in^ answering this question. An experienced 
tutor knows that^^the rules about permissible characters in 
file names vary from system ^o system and might' be expecting 
such an enquiry about file.r>4mes from, a non-naive student. 
The faci that tRis 'stiudent chose the term "special 



charactai^s" , n6t mentioned! in the text, indicates -that he 
has some )irevious experience, He certainly wouldn't be 
aBic-iag whether blanks could be stored in files, qr imagine 
that blanks come in both upper and lower case varieties. 
Thus for a system to cope with a question like this, it 
would need to have -a broader knowledge base than that 
describing NLS; it would need to have knowledge about the 
capabilities and expectations of the user, 

IndeterminaTe Reference 

Third Scenario - A similar situation (but with an 
interesting twist) appears next. 

Anl^icipating students' uneasiness . and nervousness 



before performing their first task, NLS-S.Cf: 
rather precise instruQ^tlons . To wit:^ 

tl^Lv^iOADING A FILE 




I'd like to show you ^t'F^^ frle n^amexl' DINNER so you -can 
see how an NLS file is^ structtired# 



V / 



Your first task is^to ioad t^is fi'le so ypu .can vro^rk 
with it. ' / / * X 



When the BASE C: appears /^ype>t1ie command 

load <SP> file <SP>>0rNNER <CR>^ 

te that you should" terminate each word of • the 
ommand with a space (<SP>)^ you should terminate 
e entire, command with^ a carriage return (<CR>). 

(You may type DINNER in either upper or lower 
c^ase -letters /) ' , \— / * 




As ti^rs single comman/1 compWtes the task, when the 

\ 



neiTt BASE C: appears /type 

quit <SP> <CR> 



1^11 then .check whaJt you've done ' and poinx ;.j:>if6 \any 
mistakes ' you may have made. Please be sure you type a 
<SP> a|:ter "quit", before you type the <QR>. 




s ta.sk, you 
character , or 



If 'you make a\ typing er^ror while d<5^i 
may" use, <CT^L-A^' . to remove- "the 
<CTRL-X> to delete /the entire line. 

These commands work in the same way whether/" you're 
• " V typing .to me or^to NLS. ^/ ^ 

Do you have any questions befoi?e doing this task? 



And^here" the user as'ks 



"Do I type the entire command?" 

This is a situation^ in which even a human tutor might have 
^difficulty figuring out what this user wants to know. Let's 



consider some of the possible answfers.* 



/ 




1) Focusing an the Ijast two sentences before questions are 
invited, the system coiild reply ^ 

"no. you delete the last character using 
the <ctrl-a> command, 
procedure: you depress the .ctrl . ^ ^ 
key and the a key simultaneously.' 

This is^ NLS-SCHOLARese for "Nq. You donj^^^-h^e to spell 
out <CTRL-A> to delete^ a charact^. You only have to 
depress the CTRL key .^d the A key simultaneously." 

2) Focusing on the ykfhird and the fourth sentences, the 
"^System could Xans'wer with something akin to "Yes. You 
. must spell out the entire command exactly as you are 

/ told." 

3) Finally, the answer could be directed to the fact that 
all parts of a comm^d must be speci-fied, and to typ6 
only the first part of p. two part command leads nowhere. 

7 

The twist is that jCt\^ user model in most people's mindj 
would not be sufficient to identify the purpose of 
question •^'^'Whfy wovild anyone ask it? Indeed, isn't the nts^nneir 
in ^.^wiich /the commands^/ for the first task are to be typed 
cfearlyydescribedt^I^'t it self-eviden t'/that all parts o^a 
comman/i ^^ie^^ \>^/A^^(^xt'i^d before i/ can be executed^And 





haven^'t^udents already used the <Q 
luction? 




command /in the 



The solu to this riddle is that this < particular 

questioner was familiar with NLS and was accustomed to 
typing just the first letter (or two) of each command, using /J^^ 
NLS's expert input mode^ His question^ef lected his doubt 
that NLS-SCHOLAR really mearrt^i^or^i to type each and every 
character of a cojmrsTi^ and wanted the system to confirm its 
insJ:u?Mjtrfions . This familiarity can be gleaned from watching 
his performance on subsequent tasks, but not at the time tl 
question was asked, just before the f irst^,..-trask in Lesson 
One! , 

It might^be argued that the needed information could 
have been obtained from a user profile .collected beforehand. 
The problem of acqyaring it might be handled by inserting 
quesitioris ^ into/ the introducti^m'^''^ about his previous 
experiences. yOne could find oux, for example, whether he 
was familiar with^^^t^i^inals , computer systems (if so, 
whether TeNEX op- others), editors (if so, whether NLS or 
others, and if NLS, which ver^sion) etc . If his -answers"^ 
warranted it, certain parts of the introduction might be 
skipped; a fairly detailed user profile cou44 be generated 
from this information. 

A limited user pr^ofile could ^ be easily gathered and 
should be of assistance in coping with questions like the 
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above, but' using it in the way we have described implies 
that the requisite'^lcnowledge .about other computer systems^ 
terminals, characteristics of user behavior, etc, will all 
-have to be within the system's knowledge domain This 
multifold ex^^sion of the system's field of expertise and 
its integrXtioff in^o a coherent whole, would be a formidable 
undertak^Cng, 




It may be argued that the adjective "entire", appearing 
the fifth sentence of the tutorial material and in the 
tion, is a clue that helps to link the questj^h with the 
desired answer. As mentioned earlierj^a^'^other context, a 
person involved in a jiaJ.-e-gtT^often adopts the same w(5rds'' 
that ^jflfer-^^ust used by the oth»^r party, ^^Here^^-tife n , we have 
a possible way out: lexical clues can help disambiguate what 
a student's question is abodt. But that won't help us 
sufficiently as the. next Bc^nario will show. 



\ 



Fourth scenario - After having learned how to use. the Delete 

^ / ' 

command, and after having actually prac t iced *J:KHe command by 
deleting three statements m his own workirvg' file, the user 
is toldi ^ 

Please print the modified DBiNER fil.e so you can see 

that the statements containang "tomato", "rhubarb", and 

"strawberry sbortcake" , have all been deleted from the 
DINNER file. • 7:^. . 
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After he prints what he is asked, the system continues with: 



Note how the statement numbers have been changed by 
NLS. You can see that many statements have beer 
renumbered ( 'promoted ') , some of them acquiring t' 
statreraent numbers of the deleted statements • 



Although s'bfc^tements 1A, 3A1 , and 3B were all 

deleted, these statement numbers still exist in our 

file — but the statement contents are now 
different . 



Would yoa like to ask_any_qtLeatlon-s^ - - - - 

At this point, the user asks: ' 

"Can I delete these modifications?" ^^-^-^ 

_ Since many people find it hard^ to understand this question, 
let us j^arify it with the help of a paraphrase 

"Can I pe^tore the contents of the file to what they 
were before anything was deleted?" 

-^21^^-^. Several new problem elements are irvtro^o^ced into the 
picture by this scenario. 

In the first place , the « anaphoric reference is to 
previous actions undertaken by the student (or on the 
student 's behalf) using NLS. The reference is directed 
neither to concepts explained ^earlier , nor to anything 
represented in the semantic network (the question is not 
"Can I delete modifications"). This illustrates the need to 
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/ 

bring into focus the )^istory of change^ (modifications) made 
to thje user's work file, which is not hard to do in our 
system* 



In the second place, here we have ^a case where 
"modifications" could be • misconstrued as being inspired by 
"the modified DINNER file" in the tutorial material. In' 
reality, "modifications" . for this user turns out to have a 
much firmer root: experienced NLS users know about the 
."modification file" (a file where all the changes made to a 
. -working file are kept until the working file' itself is 
updated) and how to manipulate them. This user is not 
naive: he knows that NLS provides specific ways of 
"undeleting" and he is simply and benevolently testing how 
*much NLS-SCHOLAR knows about them. 



^ In the third place , we have the rather incongruous use 
of the verb "delete" with the obje^' "modifications". All 
. that the student has learned up to this poiat indicates that 
"deleting" is a positive action resulting in something being 
eliminated from his work file, but here deletirig something 
would result in the reappearance of that which was deleted 
earlier! If we know what kind of "modifications" the student 
is talking, about, we can make sense out of the question 
without too much regard to the verb used (try, for example, 
"restore" or "undelete", or *?do something^ about/'). 
!^ore, here we have a case where what the studei^ must 
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be speaking about outv/eighs other interpretations stemming 
from his choice of words, such as "Can I- delete (the 
statements containing^ these modifications''" 



F ifth Scenario - We begin at a point where ^e system has 
just .taught the student how to ^load^'and print a particular 
file, an the. student has successfully performed two tasks 
requirinf^ him to perform these actions. The student then 
has available the following printout of the contents of the 
file., 



< TUTOR, DINNER .LNLS; 1 , >, U-SEPT-75 13:^3 LAC ;;;; 

1 SOUPS ' y 

1A tomato 
IB' vegetable 

1c cream of mushroom / 

2 HNTftEES ■ . 
2A fried ehicken 
2B prime ribs ^ 
2C scallops 

2C1 broiled 

2C2 fried • ^ 

2D salmon 

2D1 with cream sauce ^ 

3 DESSERTS, /' 
3A pie • ' ^ . 

.3A1» rhubarb. 
3A2 blueb^^ry 
33 stravrberry shortcake 
3C ice cf^ean 

3C1 blueberry 
. . 3C2 naplenut 
3C3 chocolate 
3C4 coffee ' 
3C5 peppermint 
3C6 cherry 

•The system ^^ins. to describe this file as follows: 
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THE ORIGIN STATEMENT 



Let 's" look at the information in the file. 

Notice that there is a line at the top which rives 
identifying: informat;ion about the file 

This line is called the 'orij^in statenent' nnd is 
supplied by NLS. 

First it gives vou the name of the 'directory' (a place 
in the memory) in which this file was stored. Then it 
gives the full name of the tile^ and the date and time 
of its creation. 

The file name includes an 'extension' specifying 
^what kind of file it is. 

In this case it says that this is an "LNLS" file. 
(LNLS stands for LISP-NLS and indicates that this 
file was made by our LISP implementation of NLS.) 



The number afKer the file name is called the 



And the student asks: 

''Are the brackets part of the statement?" 

Here we have two "ananhoric references ("the statement" and 
"the brackets") and a questioned inclusion relationship 
between them* 

Finding the referrents (the first line of the printout 
as a realization of "the statement", and the left and right 
an?le brackets within it as "th^^brackets" ) involves methods 
of solution not required previously. "The statement" can 



'version 




The "1" here indicates that this is the first 
version of the DINNER file that's been made. 



Do you have any questions? 
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readily be assipned the referrent "origin statement" by 
means of the^previously hypiothesized representation pf the 
tutorial mater jf^a\^and b^ focusing, but from there on we face 
entirely new ^prol4em3. In the first place, the student uses 
"the brackets" ^to describe some portion of the content of a 
statement. Surely we can not expecjt^ the system to be 
capable, "in general, of dealing semantically with the 
contents of user f^iles^. In fact, referring to ^^atemen? 2D 
as "the fish" is possible only because of our knowledge of 
zoology, which has little to do with text editing systems or 
with NLS in particular. 

Secondly, although "origin statement" is a perfectly 
valid referrent for "the statement", i/hat is really neant is 
"the particular realization of an origin statement that is 
representejd in the first line of the print out". 
Presumably, quite a bit of . inconclusive inferencing will 
have to go on before the system quits trying to find a 
connection between brackets and the concept of an origin 
statement (after all,,, ^square brackets can be used in file 
names ! ) 

In the third place, even after the correct referrents 
have been identified, what sense does the question have? Why 
shouldn't, a part of the content of a statement not be a part 
of t^e statement? Isn't this obvious? And if so, why would 
such a question be asked? "If the interpretation "upper case 
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blanks are treated the sane as ^lower* case blanks" could j^e 
rejected for beins^ 'trivial why can't this ope be rejected 
similarly? 

The truth is that we don't know why this particular 
user asked the question. We can only speculate that he was 
a TENEX user and was wonderins^ if the angle brackets were 
used' in a fashion similar to the way directory names are 
denoted in TENEX; or h^e may tiavef been prompted to ask this 
question because of the way NLS-SCHOLAR denotes certain keys 
(<CR>, <CTRfc-A>, etc. 

This is a giod place at which to stop and recapitulate 
the preceding analyses and discussions. We hare^s^en how 
each scenario has introduced new orpblems, and how each new 
problem has required pore and more complex methods of 
solution — and yet, there is no indication that this 
escalation of complexity has ceased. 

Proposing those methods, we stretched available ones 
and hypothesized new ones to such an extent that continuing 
to do'so would have been utterly unrealistic. For example, 
the user models we require would have to encompass a large 
amount of "world knowledge" in order to cope with situations 
such as the ones exemplified in our scenarios, and yet the 
theory underlying such models is in its infancy at best. 

The exercise we engaged in is certainly useful and 
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illustrates the need for continuing research, but above all 
4t^ demonstrates the need for a pragmatic approach, i,Q,, one 
based on accepting the seriousness of the difficulties and 
finding a way' around them. Rath^ than exploring a large 
number of ^plausible interpretations of a user's request, it 
Is better to either forestall -the request, or to seek its 
clarification. 
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^SECTION IV - RECOMMENDATIONS AND CONCLUSIONS 



In this section we summariTz^-. conclusions reached for 
the ^ most .part in previous sec tionsT^^^^^'B^nd we formulHe 
recommendations for further work. . Our content ion^^ls that 
one more year of relatively— low. level effort can male 
NLS-SCHOLAR a very useful operational system. 

Our first recommendation is to continue to improve the 
Ent^lish front end module to rid it of the naf^^ing little 
problems described extensively in the pi^eceding^ sectionT 
This can only be done on a continuing basis, correcting the 
problems as they appear in the course of bona fide usag^^^of 
the system by the type of users for whom it is intended^ 
This process will be long, but tne result should b^^^'a^""*^^^^^^ 
able 'to answer as many as 80% of the reauests posed* In 
parallel with this effdrt, techniques such as the 
sketched in the previous section for circumventing the 
harder problems should be developed and tested, and research 
efforts aimed at attacking these problems head-on should" 
stepped up. 

Our second recommendation is to improve the task 




evaluation module in the following ways: 



gSk! 



-1) Make it point out more clearly what is wrong vkTtTT-^-a^ 
student's result. For example, when this module responds 



(jO 




v 



"I wanterf you to change A into B but you changedv A into C"j 



_ it is hard to see sometime^ what the difference between ^ B 
a:nd C is-./.^-In other '^ords, in our efforts to avoid 
presenting the offending text in isolation without 
conte^xtual information, we^ went too much in the other 
direction; we showed so mu^ 
specifics got drowned! 



hat the 





2^ Augment the existing £asl< -entries in the 
network with a list of expected errors and specific way^a to" 
^repoj^t-^HienK^This v/otJld^ permit by--passTni^--^t1ie"standard 



reporting format rf\one of feh^se-;;«^exfific ^errors were found. 



3) Implement 



Let mexfix it" f^acili 




ostly consequences of the tasK ev^yators s ^e 




•^void the 



his facility 57ill hand back a task^ environment to the use 



^'^;;vj^fter the ^ system has found fauiT?^itfi it and has require 
^Ihe user t^^ do it all over a^itf^n. In way, ui^s' v^hat 




;r^c^Ilze what is N^ron^ and what i^'*t^xl>^cted--TTf'^^ coui< 



patch\ up their work' and satisfy the . ta^l^ evalua^^f 's- 



require^n^^nts in their.^ownr way . 




Finally, 
"intelligent" 



iTId ^r^^e^ally make this " 

give i^fe\the ability, te underhand 
tentions 'and to offerx helpful 
to point outN.what is wrong "-^fith 
a resul t ^^he most .h^Tpful ssi^uation is one where the user a^;^ 
solution methods are scrutinized and critj^^ized .-^This area 



and interpret the 
comments. It 




is certainly 'one where further developmen^ is needed. 

Our third recomraendatiorT * addresses the tutorial 
material, ^Although it is certainly in good shape, it could 
be improved by addins^ the capability for the user to 
j'edirect the order or'^lg^sentat ion of a lesson via requests 
su< 

"Let'>-go back to'-I^&bETINS BRANCHES" 
"Tell m^_agaiir-a^out •<CT,RL-X>" 

We have the >v^cessary"^oundwork ^o handle these -requests 
for review, Th^only H^rcrfei^in is 4?ow to apprise the user of 
the new content h^iXto i^Qr^^^:^;^^ 
fulfilled; that ^l^sjh^^p<^p indl:^.^t^M^race^*^^ 
has been restored co^'^aii^^^^?-]^^ earxW.er incari^^ion. 





^Ve could also"^hii-iidle re^iiffe^s like 
"Let's skixp this task" 
without too much ^diff iculty . Here the^vece^ssary chanj?^s to 
the .^,jjser 's file, to brtnp it t'o^ha^stateMt would have 
acquired after th^ task had "been completed , ^must be 



explained and justified* Requests of the form 



Let>s skip all about INSERTING" 



and 



a larpce forward 



"Teach me about VIEWSPECS" (implying 
jump) 

raise other^ Issues as well. Not only 

bringStpg the file up to da^te more complex to^xplain as many 



the prol)lem of 
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tasks'" may be involved^ but al:^o some of ^ the concepts and 
terminology skipped ov^er niay be neec^ed by the student i|i 
♦Order to comprehend the following material.. Allowing the 
student to ij|^view is relatively easy; allow^ing him to skip 
fc^rwar'd is ouite difficult ^iven the linear 'development ^f 
tlie textual material. 



EpiloRiue 



It is easy to ,jump to the conclusion that * the 
unresolved problems we have "dealt with so extensively, 
preclude syste.ms' such as NLS-SCHOLAR from becoming useful in 
an operational environment. This » conclusion would be 
errdtieous for several reasons: ^^^"""^ 

a) The frequency of occurrence of "hard, problems /^'i^s^'^ery 
• small » Most > ©f the users' requests we have seen belong 
tp the "easy to answer" ^category , regardless of the 
actual performance of the pr^ent version df NUS-SGHOLAR. 

2) As more and more of Vhe little problems are irpned out,, 
users will— he positively reinforced towards expressing 
their requests^ in the kind of English the system 

* understands , ^ and with the precision of formulation the* 
system requires. ^ * > v * 

c) As the^>iumber df failures * decreases and^ the number of 
user's increases, it hje^mes both feasible and economical 
to provide a 'human expert to back up the system as a |<ind. 
" of "consultant". In a computer network environment, many 
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users from different si 



immediate , and Dost eff 
T 

that while humkn experti 
h 

of one expert at any one time^ 




iRe of this 
^Notice also 
concentr^^ted in the hands 
)erts located in many 
sites can take turns at mindins; thif system;, i»e,,' human 
expertise may be concentrated bu/ not centralized^ 



Waiting until "intellicrent". pal systems becoijeT ^Spable 
of 100J stand ^alone operation JLs*-*'*'^th fotfile • and 
counterproductive. It is fuj^i'le because tha^ ^'kind of 
performance is probablj^j^lfllpossible to obtain /just think of 
how few people can do itl). But, more impo/tantly, it is 
counterproductive because widespread use o/ an 80? effective 
facility, for. example, would multiply by jS very lar^e factor 
the consulting capacity of a human exEfert, enabling him to 
reach more people t^han he could otrter\?ise and to address 
himself to the relevant problems quicllly. 
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DIX 

w of NLS-SCHOLAR by ISI 



following ^evaluation report was written by David 

Wilczynski, of the Information" Sciences Institute of the 

University of Southern California, at the specific request 
of the Contracting Agency^ 



I. INTRODUCTION 

/ 
/ 

This review is based on my ov;n experience in early August 
1975 with NLS-Scholar, a mixe^-ihitiative tutorial CAI 
systejn for teaching a basic subset of the text * editing 
subsystem of SRI 's NLS programming system. 

NLS^-Scholar , programmed in INTERLISP, was written by Mario 
Gr4gQetti and his group a^\RBN^ The system has evolved from 
Jaime Carbonell's Scholar \(which teaches South American- 
geography) tagether with suo^tantial influence from Brown's 
SOPHIE system, Thip system 'i^r organized to-: 

a^ Present t'eiclrual, tutoirial mateni^al t.o introduce the 
user to a terminal and to NLS. 
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b) Provide a simulated NLS system to the ,us6r on which to 
^practice what he haS learned, as- well ^fs to do 
system-generated NLS ta^ks. 

. c) Provide a natural language question-answering 
component which responds to user queries jby: 1) doing 
AI~like searches in its fixed data base, or 2) 
' "executing" the right NLS commands on the user's current 
file to answer dynamic que,stions. 

d"^ Present various NLS tasks to the ^"^user to test 
comprehension of the material just presented. 

The course is divided ^nto the following lessons. Each 
les3on takes, "about 1 hour, with-' many variables determining 
the exact length, load average, attention span, competence, 
etc . 

Introduction - Control characters 
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Lesson 1 - 



I 



Conmands: Load File, Print File , -Delete , Update 

Concepts: NLS f41ee, MLS conmands, NLS^pronpts, structure 
units (statement, branch), string-' units 

liesson 2 - . 

> 

Coinmands: Print Rest, Jump, one-character commands 



Concepts: Control Marker, content addressing 

Lesson 3 - / 

Commands : Insert , Create File , Subr-ti tute 
Ct)ncepts: Level, level adjustment 

Lesson 4 - 

"uomnanas : Frii/: , Transpose , Move , Copy 



Concepts: p]/exes, Grouos 
Lesson 5 

' Commary^s: Show Viewspecs , Set Viewspecs, Reset 
Views^ecs, Output, and Help, 

Concents : Viewspecs , Text File 

II- /general impressions 

vS> is well suited for CAI methods; NLS concepts are short, 
factual, an4 "nonphilosophic , " a g;ood method, is available 
for testinr competence (either interfacin'^^ri r^citly to IILS, 
or sinulp tin? it \ and the ^in fornation ir, incremental and 
additive rather than diffuse* ^ 

The main point is Scholar did teach ne NLS. At the start of 
the program I knew nothing about NLS other than what\it is; 
now I know the NLS terminology and how to use the . syadtem. 
However, improvement is necessary in several areaa^ if 
Soi^olar is to be a finished oroduction program, competitive 
vuth pos-sible alternative teaching methods. The following' 
two sections will review Scholar's strengths and* weaknesses . 

III. THtC KMVIRONMENT OFFERED BY SCHOLAR 

The, Scholar CAI system is classical in that text is 
presented to the student in prearranpted frames with tests 
usually following each.^ The inclusion of a natural language 
interpreter is an innovation which allows the student to ask 
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questions during the program. It turns out that this noi*e 
of operation has advantages for nonstudent types. Studies 
have shown that people relate well to computers, suffer less 
anxiety, and feel freer to experiment and ask Questions in a 
CAI environment . The critical aspect of such a system is 
its transparency. 

If the student notices (or becomes preoccupied by) the CAI 
machinery, he can \perf orm in the short term (answer 
questions , do short talsks ^ , but lacks global comprehension . 
Thus the type of display and the "smoothness" of the system 
become important fact,ors for people not used to operating 
such devices. Specifically, NLS-Scholar is intended for 
typewriter terminals. Having written a CAI system for such 
ter^^inals myself, I have verified that all students are 
acutely aware of the typing noise and slow speed. I used 
NLS-Scholar on a 2^00 baud video termiWl and was much more 
satisfied with tJ^e results. Since Uijere are times when 
hardcopy is needed for back referrals, BBN would do well^ to 
offer the appropriate hardcopy text to ,the student as an 
a<^dition for t(ie ;video terminal. 

A parameter of system smoothness is its response. veness . A 
high load avenage (virtually anything above 4)' combined with 
the slowness of INTERLISP made Scholar move at an unbearable 
crawl\ When, the load average cane down to 1 or less, the 
system moved about sprightly. The difference here is more 
than' one of .convenience. .No user (unless he is forced or 
paid ^ will sit through a session of -Scholar on a machine 
with a high load average. If he nust, it v/ill turn out to 
be a painful, wasteful way to learn NLS. \ ^ 

\. ** 

A few of the INTERLISP features caused some unnecessary 
distractions. I found the rarbaf^e collect nessares ("Excuse 
me, while I rearrange my memory ! '*)^ disconcertinf^ since they 
caused a visual break in my concentration. I appreciate the 
attempt to explain the impendinr^ delay, but I think the 
typed messa.i^e is too visiBl^ 

The preprocessing of all questions and responses by ^DV/IM 
also caused some anusinp incidents. For example, in 
answerinp the Question, "What character prints the context 
of the CM?", I responded DWIM turned the slash into a 

(a common INTERLISP occurrence) and then NLS-Scholar 
told me that not was the corr.ect answer. Those 

sort of bugs are not serious and easily repairable, but must 
not exist in a released" product . 

NLS-Scholar offers a medium which can be started v/hen 
desired ' (assuming machine availability^, stopping at 
arbitrary points, and proceeding in^ whatever pace Is 
comfortable. If ^ILS-Scholar were ^ set up to operate at 
different modes .(beginner, expert, reviewVthen the problem 
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of retraining and refreshing; , previous NLS users v/ould be 
simplified. This nay not be a simple addition to make in 
NLS-Scholar, but judging from discussions v;ith users at 
-ilun^terAF? , it would be powerful and useful. 

IV. ^ TEACHING COMPONENTS 0? NLS-SCHOLAR 



The three main components of. NLS-Scholar are: a) the 
tutorial Information, b) the natural langua.f^e interpreter, 
and c^ th^ test manag^enent. vThe. first and third are CAI 
standa/^ds , while the second is in the realm of Artificial 
Inteld' ipence ^AI ) . 

4)^ The text material was impressive; it was presented 
concisely and accurately. At no time did I feel that I 
was being either overloaded or nursed throufh, both 
factors which led to effective and willinr: comprehension 
y of the material. It is easy to overlook or underestimate 

, Quality in this area because good tutorial services are 
not as visible as poor ones. Because of this phenomenon 
I want ^to emphasize^ the exceli^^ce of the tutorial 
* information . ^ 




The natural lanpuape interpreter is more complicated 
to evaluate. Most likely, it is the most complex part of 
the system, yet orobably the least useful in its present 
form. The main problem concerns i^s robustness. O-ften :1 
asked^^^^a— -^mple question like "Please. review tJhe 
'tSffe -character commands and p;ot only a "Please rephrase 
your request" reply. In this case I think the problem is 
that "review" is not part of Scholar's dictionary. 
Hov/ever, in rephrasing the question ^o something .like 
"Tell me about the one-character commands," I would just 
ret a list of them without functional definitions. To 
get i.'hat I wanted I would have to ask for each 
individually, (e.r., "Tell ne alJbut the command."). It is 
-disconcerting to. have the parser or retrieval mechanism 
fail on a simple request, but not to know why is worsi5. 
Just asking for a rephrase does not indicate what the 
failure was; this information will surely be useful in 
composing a different reouest. V/hether most users would 
want such infornStion is a different Question; I would 
have likjld it. 

It is hard to be critical of this natural languafe 
business, since the problem is still a major research, 
not developmental issue. ' Still," I wonder if Scholar's 
interpreter is state of the art; I am thinking of Woods' 
moon rock program. Since that propram is also a BBN 
product, it would be interestincr to get. a comparison of 
the two systems from, the ULS-Scholar group. 
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The lack of robustness of the English, interpreter 
detracts somewhat from Scholar; I iTound myself not using 
that component. The tables look-up kind of questions it 
could answer, would be better solved by just having access 
to the table in some primer format, ftgain, the lack of 
field 'testing may indicate that this is just a personal 
reaction; but the shallow range of questions and answers 
, pakes the 'current worth of 'this subsystem suspect. 
Certainly, it doesn't fulfill the capabilities of a human 
tutor. 

C) The ^test management phase of Scholar is composed. of a 
series of questions which are answered either by doing an 
NLS taskr or talking directly to the Scholar top-level. 
In both cases* the answers or performance are evaluated 
with feedback as to correctness^ The ability to check 
answers is one of the more difficult tasks for a CAI 
system when the domain of true-false or multiple choice 
questions is not used. Scholar does admirably here but 
is far from perfect. 

The top-level type questions, (e.g., "What is the 
statement number of the statement that will be printed if 
I now use the backslash command?") will be looked at 
later. The NLS tasks,, the heart of the testing 
component, will be reviewed in depth* 

The basic mechanism for matching a task answer to the 
correct one seems to be: 

a) If a file manipulation task is involved (e.g., 
INSERT, DELETE), then the resulting file and the* CM 
(control marker) are checked against Scholar's 
expectation. 

b"^ If n printing task is involved, the output of the 
print command is trapped awd matchjed against expected 
print, and the' CM is 'checked for positioning. 

At no time does it appear that Scholar looks at ' the 
student's input sequence. This lack leads to many 
unfortunate experiences. For example, one task asked 
to delete two consecutive statements, expecting the 
user to use the sequence, "delete statement 1B5, delete 
statement 1B5," to account for th6 renumbering done by 
NLS. I tried, "delete statement 1B6, delete statement 
t35," to accomplish the sane effect. Scholar told me I 
did the task correctly and then the next frame 
described how I could have accomplished the same task, 
by deleting statement 1B6 and then deleting statement 
* 135 . Not serious , but the question of system 
transparency arises. 
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A more serious flaw in this purely extensional fijirm of 
testing appeared in the tafsk to '^test the use of 
^CTRL-e;> for insertinp; a series of statements. I did 
the task by insertinp all the statements at the same 
level (superfluously usinp; CTRL-E after each insert) 
before ^oing back to insert substatenents . -Even though 
the resulting file was correct, the CM was not where 
Scholar expected it and so I - was informed- of this 
"error" and told to redo the entire task from scratch! 
Needless to say, I didn't enjoy retyping the whole 
thing. Worse, however, was the failure of Scholar ,to 
recognize v/hat I did, tell me the right way to do the 
task (i.e., use one CTfiL-E an^ move up and down levels 
using the L: prompt) and then let me proceed. It is, 
however, easier to be critical^ of this flaw than to 
suggest an alternative. A deep unde^standing of the 
intensional comnand strings represents a large (perhaps 
unknown^ e.f f ort . If accomplished, there is no Question 
that the system will appear much more intelligent .then 
it currently does, as well as being more useful. 

Other examples of situations where this type of problem 
come up gan be given, but are not necessary to this 
review. Some of the techniques used to check top-level 
questions ( those not requiring * the NLS simulator) are 
also open to improvement • For example, one question 
.expected CTRL-X as the answer; I typed <CONTROL-X> and 
was tol;d.^I was wrong. Another time I answered a 
questr<fn' with LIN£-FEED and Scholar wanted <LF>. These 
two cases should not be constrtied^as nitpicking, but as 
an attempt to point out situations which make Scholar 
seem less suitable as a training method than standard 
teachi^/^ nethods. Too many of these trivial flaws will 
discourage the CAI user. 



SUMMARY AND COflCLUSIOUS 



As I mentioned before, it is much easier to point out flaws 
in a CAI system then to recognize its Qualities. Experience 
with standard nethods give rise to expectations which are 
then used \o judge CAI systems. .Yet, criticisms of Scholar 
should be tempered by one observation. Scholar does teach 
the student NLS effectively. Assuming that the lodal bugs 
in Scholar are fixed (a few have been described in this 
paper^, a useful system exists which can be used to br^TTh, 
potentia l NLS users. ^ 
X- 

Still, changes can be made which might expand its range of 
use as well as improve its performance. Several hav§ been 
pointed out in this paper, for example, making the natural 
language component more robust, adding analysis of the 
user's input to the current extensional analysis,, endowing 
Scholar with other training modes, expert, review; .etc. 
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None of these possibilities are simple; more fJLeld t^e^tin?: 
is necessary before firra conclusions can be made one way or 
the other. Yet, once Scholar is raade more complete in its 
coverage of NLS, it will be a viable product and should be 
evaluated as such by af:^encies interested in NLS. 

Som-e pvirely system questions also need addressing. Can 
IfCS-Scholar ''be a viable product as an INTERLISP program 
(thus bound to TENEX)? Are there enough machines with enouph 
time slots of low load average to acoommodate the potential 
Scholar users? I am sure other questions of this type will 
arise if research into Scholar is continued. 



I 




77 



73 



Comnehts^^j^pa .the review ^^.^^ 
^ by Mario C. Gripnetti 

It seems to me that the review is, overall, a rather 
positive one, MLS-SCHOLAR seems to be able to do it^ most' 
important job, i.e,, teach NLS. 

Many of the problems that Dave points out are trivial- 
to take care of: garbage collection raessapes , DWIM's 
bu3y-bodiness in unwarrantedly exchanginf' for and 

more ways to represent CTRL-X or <LF> than we ant icipated •/ 
After all, the main soal of the field testing perform! 
under this contract was precisely to bring up these kinds ol 
problems. 

Dave is wrong in his assertion that "at no tine does it 
appear that Scholar looks at the student's input sequence":, 
TI>e systen does look at the student 's actual input when he 
an^s^rs questions. The fact <that Dave's clever answer 
(delete statement 1B6 and then delete statement JJ^i^^was not 
handled Intelligently was due to a stupid bug in one, of the 
predicate functions in our answer evaluation module. Again, 
this is .a case in pointy for the usefulness of this type of 
testing to xTfe system's designers. In general, however, 
Dave's criticism is v^lid: when the student performs a task 
using NLS, the commands he .types are not^ looked at and only 
their consequence'B'^^^''*a^ used to evaluat^^ what he has done. 
We'd like very muclg, to tackle the difficult problem of 
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int6irrti<5i%6j^ cc^prehension ; ' if solved we would have a much 
smaifter sy§1>«iii! \ 



J 



Other difficulties^ referred to in the review are more 
serious. Indeed, we need to provide feedback as to why a 
reque^^fails to be understood. We had wanted to tackle the 
problem of parti^,**»r6omprehension and try a few strategies 
that ap^ggj&t^'proroising. However, the pressures arising from^ 
C^'mited time arxl^^ resources , and' the purely developmental 
type of work in which we have had toMconfif^e our efforts, 
precluded the performance of sorely needed research work. 
With respect to our use of "Wood's moon rock program", this 
is another thing we've kept on the back burner for some 
tffire-*.^, Jiowever , it is questionable that Just a more powerful 
parser would have made a lot of difference in the system's 
ability to respond to student's rjequests. The difficulty 
here resides not so much in tj>e literal interpretation of 
questions as if they were precise formulations of the 
specific bits of knowledge the questioner seeks, but rather 
in figuring out what each particular student may have meant 
to ask;, pi^^en where he is, his previous performance, what he 
ought to know, what he seems to have learned, etc., etc. It 
is surprising to see how many questions are unanswerable, 
even to a human, when taken in relative isolation. 

Finally, a word about, efficiency . We do not think that 
3 ^ CPU minu£es per hour is a terribly inefficient and 



unacceptable way to administer a CAI lesson. We agree^ 
hov/ever ''and v/holeheartedly ! ) with Dave's observation that 
v^hen the load avera<ye in a peneml ov>^pose t ime^sTial^inr 
•vstem such as^TEN»':X reaches about 4, it is better to ouit 
and hone. This is not a problem that affects NLS-SCHOLAR 
alone; whelv^a large system such as TKNEX is saturated, 
nobody gets anythiYrg-^done efficiently, including NLS;users. 
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